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ABSTRACT 

A test bias definition, applicable at the item-lev>l 
of a test is presented. The definition conceptually equates test bias 
with measuring different things in different groups, and 
operationally equates test bias with a difference in item fit to the 
Rasch Model, greater than one, between groups. It is suggested that 
the proposed definition avoids confusing etiology with measurement by 
capitalizing on tL3 "objectivity" property of the logistic Rasch 
Measurement Model. Application of the definition, to 9m applicants 
(black = 367; white = 547) in a "real" selection situation is 
described and resulted in identifying two items as biased. The two 
items so defined, were different than the two items identified as 
biased by comparing the item success rates (i.e., item difficulty) of 
black vs. white candidates. A content evaluation of the items by two 
black, female reviewers was subsequently performed. Their comments 
lend preliminary support to the proposed psychometric test bias 
definition. Additional encouraging support is provided by the match 
between the content comments and the item bias index values, for 
other items in the test. Implications for future applications and 
research are presented. (Author) 
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Psychometric definitions of bias vary on three dimensions, each a 
dichotomy, for a total of eight possible cells in the classification scheme 
(Durovic, 1975)' These dimensions include the score-level addressed 
(i.e., item vs. test), th3 role of an external criterion variable, and the 
role of external value judgments. The dimension important to the present paper 
is the score-level addressed; only definitions at the item level are considered. 
The present paper is concerned with- one of the eight possible cells: 
item-level definitions with no external criterion variable and no external 
value judgments needed. 

Test Bias Definitions at the Item Level 

Few test bias definitions are addressed to the item-level. The 
simplest definition, at the total test level, equates test bias with group 
differences in test score meajis. For the item-level, it caji be modified to 
equate bias with group differences in item score means. Test constructors 
have xraditionally used this approach to eliminate those group differences 
they consider irrelevant to the purposes of the test. For example, items for 
the Stanford-Bine t were chosen, in part, to eliminate score differences between 
sexes (Anastasi, 1968). However, this approach to test bias is faulty, since: 

tests who^H items have been selected K'ith reference 

to the responses of stny special groups cannot be 

used to compare such groups (Anastasi, 1968, p. 179). 
Critics of the mean-difference definition argue that, if the test accurately 
identifies existing differences in the ability being measured, then differential 
group performance should not label a test as biased. To do otherwise confuses 
etiology with measurement (Anastasi, 196l). Conceptually, test bias applies to 
the properities of the instiaiment and not to the persons taking it (Durovic,1975)- 
Tr3 problem of etiology vs. measurement exists whether the mean-difference 



definition is applied to the item or to the total test score level. 

Another definition of test bias, at the item- level, is provided by 
Cleary and Hilton (1968). They state that an item is 

biased for member^s of a particular group, if on 

that i tem, the members of the group obtain an 

average score which differs from the average 

score of other groups by more or less than 

expected from performance on other items of the 

same test. That is, the biased item produces 

an uncommon discrepancy between the performance 

of members of other groups. In terms of the 

analysis of variance, bias is defined as an 

item X group interaction (Cleary and Hilton, 

1968, p. 61). 

The Cleary-Hilton definition of test bias has been used by other research-/ 
such as Angoff and Ford (1971 ), Echtemacht (1972), ar ^^oepfner ajid 
Strickland (1972). A variant of this definition was employed by Evajis and 
Reilly (l972) who investigated speededness as a source of test bias and defined 
bias as the time x race interaction. In all of these studies the ANOVA method 
is used and the interaction term with race serves as the definition of test bias. 

This definition differs from the simple mean difference definition. 
However, the ANOVA approach still equates meaji item score differences with test 
bias; it simply allows the difference to be determined by the group performance 
on the tot at of items, rather than mandating no allowable difference 
between groups. 

Objective Test Bias Definition 
An approach to defining test bias at the iteiu level, which avoids 
the etiology vs. measurement problem, may be provided by the Rasch Model 
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(Rasch^ i960). Georg Rasch developed several measurement models and one of 
these models, referred to here simply as the Rasch Model, applies to test items 
which can be viewed as a dichotomous, or binomial response situ .tion of correct 
vs , incorrect response (Rasch, 196O; 1966a; 1966b). The Rasch Model involves 
two parameters: one; parameter for the person's ability level, and one i)ar£uneter 
for test item easiness level • Some researchers refer to the Rasch Model as a 
one parameter model because it has only one parameter for test items (Hambleton 
and Traub, 1970). 

The attractiveness of the Rasch Model as an approach to defining 
test bias is due to its property of "objectivity" (Tinsely and Dawis, 1975; 
Wright, 1968) which makes it possible, in the analysis of data, to separate 
the person pareuneter from the test item parameter (Rasch, 1961). Essentially 
the Rasch Model "objectivity" property permits calitratioii of test items 
independent of the sample distribution of ability in the subjects. Thus, even 
if one group of subjects is drajnatically different from a second group of 
subjects on the ability being measured, the same item calibrations should occur 
for each group separately. That is, "objectivity" 

requires that test calibration be independent of which 
persons arc used for the calibration and that person 
measurement be independent of which items are used for 
the measure.aent (Wright, 1968, p. l). 
Therefore, it appears possible to avoid, mathematically, the problem of 
confusing person ability levels with test instrument characteristics, if we 
can capitalize on the property of "objectivity. " 

One approach to defining bias in this way, at the item level, is 
presented here. During calibration an independent test of fit to the Rasch 
Model is available for each a.tem. Since, the Model is a probabilistic one, a 



perfect fit of data to the Model is not expected. A deviation of approximately 

one standard deviation for the observed data from the expected is projected, 

that is, a- mean square fit of an item to the Model equal to one is expected 

(Wright, Note l). Therefore, test bias is c.afined here as follows: an item 

is biased for members of a group, if on that item, for members of the group, 

a mean square fit of the item to the Rasch Model is obtained which differs, by 

greater than one, from the mean square fit obtained for members of the o r 

group. By this definition, a test is not biased if each item in the tes . 

relates to tne dimension being measured in the same we "^or each group^ 

Hypothetical Illustration 

Essentially, in a measurement situation we have two groups of 

elements. One group consists of the measuring instruments, and the second 

group consists of the objects to be measured. Typically, the first group of 

instruments are test items, and the second group of objects are persons. For 

illustrative purposes, let us consider for the moment, that the first group 

consists of instruments that exert a reproducible influence readily acceptable 

as mechanical only (i.e., they "push")^ and the second group consists of objects 

readily acceptable as solid bodies. Then if each instrument, I^, is applied to 

each object, 0 , a set of rates of accelerations, A . , may be observed, 
r rj 

Then if it happened that the acceleration received by 0.^^ from 
instrument I, is twice that received by 0^ from the same instrument, and it was 
further found, aside from errors in measuring the acceleration, the same sort 
of results when applying any other instrument then it could be said 0^ has half 
the '^ass" of 0^ . Note, the mc^an-dif ference definition would label such an 
instrument, I, as bisused. 

Next, against this background suppose the mistake was made of not 
noticing that one of the instruments was strongly magnetic. If among the solid 



bodies were both' a piece of wood and a piece of steel then obviously magnetism 
would play havoc with the accelerations. The accelerations produced by this 
instrument would not fit in with the mechanically produced accelerations. If 
we did not make too many such mistakes then we would easily discover and locate 
the error. Within ;the context of the "acceleration" experiment, the instruments 
which discriminate between the metal and non-metal objects (i.e., the magnetic 
instruments) ai-e "biased" by the definitional approach presented here. 
Empirical Illustration 

To empirically explore the definition offered here, selection-test 
responses of 91^ adult candidates (Black 36?; White 54-7), for a wide variety of 
client^oriented government positions, were analyzed. A 1^ item multiple-choice 
test, that was part of a written test for the positions, was used as the test 
instrument. 

The subjects were identified by race based on a self-administered 
questionnaire. A total of 36? candidates identified themselves as Black and 
5^7 candidates identified themselves as White, at those test centers with 
self-identified Black candidates. 

Applying the Cleary-Hilton (1968) definition (see Table l) resulted 
in finding a significant interaction effect of race x items (p ^ .01) and 
therefore by this definition there is test bias (see Table 2). The items wr.th 
the largest item difficulty differences are items #11 and //1 5 (see Table 3). 

Applying the proposed definition resulted in finding two items, #8 
and //I3, with a mean square fit^ greater than one (see Table ^). Therefore, by 
this definition, only items ^8 and #13 are biased. These two items are not the 
same as the two items with the greatest difference between groups in average 
score . 
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Content Evaluation 

Next, the possibility of determining an item-content-based explanation 

for the difference between groups in fit of an item to the Rasch Model (i.e.;, 

the proposed test bias item index) was considered. There Kere several reasons 

for believing that content explanaticas might be possible. First, Rasch (196O) 

suggests this possibility when he states: 

once a law has been established within a certain field 
then the law itself may serve as a tool for deciding 
whether or not added stimule and/or objects belong to 
the original group (p. 12^). 

Second, in discussing the Rasch Model, Wright and Panchapakesan (1969) 

state that a 

source of lack of fit of an item lies in the content 
of the item. The Model assumes that all items used are 
measuring the same trait. Items in a 'test* may not fit 
together if the 'test' is composed of items which 
measure different abilities. This includes the 
situation in which the item is so badly constructed or 
so mis-scored that what it measures is irrelev^^^t to 
the rest of the 'test' (p. 25). 
Third, empirical support for content-based explanations of misfitting items has 
been presented. In a series of studies, using personnel selection test responses 
of adult applicants, Duiovic (1970) reported that the misfitting items had test 
construction defects or were measuring different abilities from those of the 
total test. More recently, Kifer and Bramble (l97^) reported that an examination 
of the items which did not fit the Rasch Model 'Indicated that about one-half 
were poorly written" (p. 2). 
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Therefore, the feasibility of revealing possible content-based 
support for the items deemed biased (i.e., items #8 and #13) by the proposed 
item bias index was explored. 

Two reviewers were selected who had no role in the preparation of 
the test used in this study an.. ^ therefore, could provide and independent 
content evaluation. These two reviewers were selected because it was felt 
they could evaluate the tests for possible bias against Blacks. 

The first reviewer was a Black female member of the New York State 
Advisory Goiiunittee to the United States Commission on Civil Rights and is 
involved in assisting the Committee to determine the extent of discrimination 
in selection practices in public employment. 

Her reaction to the two items identified as biaused by the proposed 
item bias index was immediate and intense. She had a strong, emotional, 
negative reaction to them, and argued with the premise of the items. This 
reaction did not exist in the remaining twelve items. 

The second reviewer was a Black female official of the International 
Personnel Management Association (Eastern Region) and Director of a large test 
developnent group that constructs public sector employment testa for government 
jurisdictions in New York State. She submitted written comments on each item in 
a narrative format (except for item #6, which she presented in an outline format, 
and as a result item #6 is not considered here further) which ranged from a 
succinct "seems o.k. " (Griffin, Note 2) for items #10 and #12, to lengthy 
detailed criticisi.i. 

First, her reaction to the two items identified as biased by the 
proposed definition were in substantial agreement with the first reviewer. For 
item #8, she states the item would have an emotional impact on Blacks. For 
item # 13» she argues with the item and states the key answers are not 
appropriate or even correct for most minorities. Second, she gave twice as 
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many lines of criticism to two items in the test than to any other items 
(i.e., nine lines vs. five lines). The two items which received the most 
criticism v -e, items #8 and #13, tz;.e two defined as biased by the proposed 
psychometric definition. Third, only two items received five (5) lines of 
criticism (i.e., item ffl and item #?). Interestingly, they both received 
virtually identical test bias item indices (i.e., mean square fit difference 
values) by the proposed definition. Fourth, three, and only three, items 
received criticism that the vocabulary or terms used in these items was not 
appropriate for minority group members (i.e., items #2, #5, and #6). Tne test 
bias item index, for these three items , cluster together and no other items 
have a similar index. 

While no firm conclusions can be placed on these two reviews, their 
comments lend some preliminary content support to the psychometric test bias 
decision reached by the proposed definition. In addition, the observation that 
some items, with similar content comments from the second reviewer had similar 
test bias item indices, suggests that perhaps item index differences themselves 
may be fruitful areas of study (i.e., analogous to discovering magnetic bodies). 
Conclusion 

A test bias definition, applicable at the item-level of a test is 
presented. The definition conceptually equates test bias with measuring different 
things in different groups, and operationally equates test bias with a difference 
item fit to the Rasch Model, greater than one, between groups. It is suggested 
that the proposed definition avoids confusing etiology with measurement by 
capitalizing upon the "objectivity" property of the Rasch Model. Application 
of the definition, to applicants in a "real" selection situation, resulted in 
identifying two items as biased. The two items, so defined, were different 
than the two items identified as biased by comparing differential item success 
rates (i.e., item difficulty). A content evaluation of the items by two Black, 



female reviewers was subsequently performed. While no firm onclusic.-i 3 can be 
placed on these two reviews, tiie comments lend preliminary support to the 
proposed psychometric test bias definition. Additional encouraging support 
is provided by the match between the content comments and the item bias index 
values, for other items in the test. 

Future research night consider the applicability of the definition 
proposed hero as a procedure to refine tests, identify possible general recisons 
for test bias (e.g., vocabulary problems), and examine the impact on criterion- 
related validity. For example, in the test evaluated here, some item clustering 
by the psychometric procedures of the proposed test bias definition appear to 
parallel the item clustering by the subjective content ^valuations of the 
minority group reviewers, strong emotional reaction and differential vocabulary 
useage were suggested as possible explanations for two cf the item clusters. 
Can these findings be replicated or generalized? If replicable and 
generalizable content-related explanations for bias can be found for items, 
psychometrically defined as biased by the proposed definition, then several 
practical implications follow. First, test developers could refine their tests 
by constructing substitute items for the biased ones and then psychometrically 
evaluate their effort. This test refinement process, when coupled with 
standard validation procedures may improve tests used for selection. Second, 
educators could consider setting goals to eliminate the content-related disparity 
identified by the biased items. Evaluators might psychometrically assess the 
success in achieving the goals, by administering the biased items and applying 
the proposed definition. It should be noted that this procedure is not the 
same as evaluating item success rates or achievement. If the reason of the 
bias has been removed then the test bias index should reflect it, within the 
context described earlier. Thus, if biased items are found, then, either they 
can be removed if subject discriminations on this dimension are undesired, or, 
they can be used exclusively if subject discriminations on this dimension is 
O desired. 1 1 
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Table 1 



Cleary-Kilton ATJOVA Xodel for Data 



'X = M + I, + R + s , . + IR. + IS. , V 

is(r) 1 r s(r) ir is(r) 



•Where M = grand mean 

= items (i = 



R 



r 



= race (r = 1,2; 1 = black, 2 = white) 



s(r) = subjects within race [s (r=l = 36?; 

s (r=2) = ^Ul] 
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Table 2 

Cleary-Hilton AIIOVA Test Bias Definition 



Source 


df 


MS 


F 


Item (I) 


13 


23.28 


139.99* 


Race (R) 


1 


2.72 


9.77* 


Subjects (S) 


912 


.28 




I X R 


13 


2.03 


12.15* 


I X S 


11,856 


.17 





* P< .01 
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TalDle 3 





Test Itera Difficulty 


for Each 


Racial Group 


















Black 


(B) 


v. hi 


te (v:) 


ui'. icrcnc- 




Couiat 


Percent 


Count 


Percent 


Percent 


1 


280 


77.56 


UT3 


88.08 


-10.52 


2 


265 


73. ill 


U05 


75-1+2 


- 2.01 


3 


113 


31.30 


229 


U2.6U 


-11. 3U 


h 


300 


83.10 


U67 


86.96 


- 3.86 


5 


2l*7 


68. U2 


36U 


67.78 


0.6U 


6 


212 


58.73 


372 


69.27 


-10. 5U 


7 


208 


57.62 


331 


61. 61; 


- U.02 


8 


239 


66.20 


h2h 


78.96 


-12.76 


9 


201 


55.68 


282 


52.51 


3.17 


11-" 


3U2 


9U.7I+ 


52S 


58.32 


36. U2 


12 


338 


93.63 


519 


96.65 


- 3.02 


13 


213 


59.00 


301 


56.05 


2.95 


Ih 


253 


TO. 08 


hho 


81.91+ 


-11.86 


1^" 


327 


90.58 


■ 351 


65.36 


25.22 



^iteni vie deleted 

^item with second largest item difficulty difference 



- ^-item with largest item difficulty difference 

Note-A constant should sui)tractcd from the (B4-/) Difference shovm in 
or^er to evaluate in terms of the Cleary-Hilton Definition, however in 
the present instance ^since Mi - M2<0.5, then 0.5/1^^ i^^alces little 
difference. 
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r lb o I lb ems bO 


Kascn Model for Both 


Groups 






Group 






Black (B) 


V.Tiite (V/) 


(B-W) Difference 


1 


1.03 


0.99 


O.OU 


2 


1.87 


1.02 


0.85 


3 


0.79 


l.Ul 


-0.62 


k 


0.88 


1.01 


-0.13 


5 


1.85 


1.09 


0.76 


6 


1.16 


0.55 


0.6l 


7 


0.83 


0.76 


0.07 


8 


1.13 


2.20 


-1.07* 


9 


1.29 


1.U8 


• -0.19 


11 


0.62 


0.U3 


0.19 


12 


0.88 


0.63 


0.25 


13 


2.16 


0.87 


1.29* 


lU 


1.11 


0.66 


0.;;5 


15 


1.51 


1.28 


0.23 


.All 


1.31 


1.11 


0.20 


Chi-square 
probability 
df 


0.009 

130 


0.206 
117 





^itcm //lO deleted 



^Difference "> 1.00 
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